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EMPIRICAL  BAYES  ESTIMATION  OF  PROPORTIONS  IN  SEVERAL  GROUPS 


The  problem  of  estimating  binomial  proportions  in  several  similar  but  not 
necessarily  identical  groups  occurs  frequently  in  psychological  and  educational 
settings.  A  straightforward  empirical  Bayes  approach  to  this  problem  using 
a  slight  extension  of  the  standard  Bayesian  method  for  estimating  a  single 
proportion  is  proposed.  Novick,  Lewis,  and  Jackson  (1973)  suggest  a  purely 
Bayesian  solution  to  the  problem  which  uses  a  root  arcsine  transformation  of 
the  proportions.  They  contrast  their  method  with  a  similar  approach  using 
the  same  transformation,  due  to  Jackson  (1972)  using  examples  to  show  that 
the  Bayesian  approach  is  better.  This  note  shows  that  an  improved  version  of 
Jackson's  approach  yields  much  more  satisfactory  results  than  the  original,  so 
the  advantage  of  the  purely  Bayesian  approach  is  questionable.  But  the  revised 
version  of  Jackson's  approach  and  the  new  beta-binomial  approach  yield  practically 
identical  results,  so  even  the  need  for  using  the  root-arcsine  transformation  is 
questionable,  except  when  the  proportions  are  concentrated  near  zero  or  one. _ 
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EMPIRICAL  BAYES  ESTIMATION  OF  PROPORTIONS  i::  SEVERAL  GROUPS 


I.  Introduction 

The  problem  of  estimating  binomial  proportions  in  several  similar  but  not 
necessarily  identical  groups  occurs  frequently  in  psychological  and  educational 
settings.  Standard  practice  suggests  two  conflicting  ways  to  deal  with  the 
problem.  One  might  stress  the  similarities  between  the  groups  and  use  the 
overall  proportion  as  a  common  estimate  for  each  group  proportion.  This  option 
makes  sense  if  Che  variance  between  Che  group  proportions  is  so  small  chat  it 
is  plausible  Chat  it  is  just  due  to  the  variability  uf  sample  proportions  drawn 
from  the  same  population.  The  second  possibility  stresses  the  differences 
between  Che  groups.  It  assumes  chat  each  group  is  a  sample  from  a  population 
distinct  from  Che  other  groups  and  hence  uses  the  sample  proportion  for  each 
group  to  estimate  its  distinct  population  proportion.  The  greater  the  variance 
between  the  proportions  in  the  separate  groups,  the  more  Justified  the  second 
option  becomes.  The  typical  situation  falls  in  a  awkward  middle  ground  where 
neither  way  suggested  by  standard  practice  seems  completely  appropriate:  it 
is  implausible  on  the  basis  of  Che  data  that  Che  groups  come  from  Identical 
^pulations,  yet  the  groups  are  similar  enough  that  one  must  view  with  some 
skepticism  sample  proportions  which  differ  markedly  from  the  rest  of  Che  groups, 
particularly  when  chose  proportions  are  based  on  relatively  small  samples. 

This  paper  concerns  efforts  to  find  a  middle  way  between  the  two  approaches 
Just  described  which  can  weigh  Che  direct  information  available  in  a  single 
sample  with  the  collateral  information  obtained  from  distinct  but  similar  samples. 

Jackson  (1972)  proposes  a  method  inspired  by  the  Kelley  regression  estimate 
of  true  score  given  observed  score,  from  classical  test  theory.  This  is  a 
weighted  estimator  which  gives  weight  r^^i  to  an  individual's  observed  test 
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score  and  weight  1-r  »  to  the  mean  score  for  all  individuals,  where  r  ,  is 

**  XX 

the  reliability  coefficient.  The  analogy  between  the  testing  situation  and  the 
present  problem  is  made  by  identifying  the  group  proportions  with  test  scores. 
Actually,  the  root  arcsine  transforms  of  the  proportions  are  used  Instead  because 
they  fit  the  test-theory  assumptions  behind  the  Kelley  formula  better,  being 
approximately  normally  distributed  with  conditional  variance,  given  the  true 
proportion,  independent  of  the  value  of  the  true  proportion.  Observing  that 
the.  reliability  coefficient  can  be  thought  of  as  the  ratio  of  true-score  variance 
to  observed  score  variance  reduces  the  problem  of  determining  how  much  weight 
to  give  to  individual  scores  (or  transformed  group  proportions)  to  one  of  esti¬ 
mation  of  variance  components.  Jackson  suggests  simple  estimators  which  lead 
to  easily  computed  estimates  of  the  proportions. 

By  specializing  results  of  Lindley  and  Smith  (1972)  concerning  Bayesian 
analysis  of  variance-components  in  the  general  linear  model,  Novick,  Lewis,  and 
Jackson  (1973)  provide  a  Bayesian  rationale  for  and  extension  of  Jackson's 
approach.  Their  extension  has  both  advantages  and  disadvantages.  The  main 
advantage  is  that  the  more  thorough-going  Bayesian  approach  avoids  some  nasty 
problems  associated  with  the  estimation  of  variance  components,  such  as  negative 
estimates  of  be tween-groups  variance,  which  can  make  Jackson's  original  approach 
yield  unconvincing  analyses.  The  disadvantages  are  Increased  computational 
complexity  and  some  technical  problems  concerning  specification  of  the  prior 
distribution  for  transformed  pooportlons. 

The  problems  with  the  specification  of  the  prior  distribution  of  the  trans¬ 
formed  proportions  derive  from  the  fact  that  the  transformed  proportions  are 
bounded  between  0  and  it/2  .  Let  y  denote  the  true  transformed  proportion  for 
a  given  group,  and  let  Uy  and  denote  the  mean  and  variance  of  the  prior  dis¬ 
tribution  of  Y  .  The  calculation  of  posterior  distributions  is  greatly  facilitated 
by  assuming  that  the  Joint  prior  distribution  of  and  has  them  Independently 
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distributed,  with  uniformly  distributed,  distributed  as  Inverse  chi-squared, 
and  Y  ,  given  and  ,  normally  dlstribuCed.  These  a  --umptlons  are  Jointly 
compatible  only  if  the  prior  for  is  tightly  specified  to  keep  the  probability 
of  large  values  of  negligible.  This  is  no  problem  if  a  tight  specification 
accurately  represents  the  investigator's  prior  beliefs.  However,  Novlck,  Lewis, 
and  Jackson  caution  that  setting  an  unrealistically  tight  prior  in  order  to 
make  the  mathematical  machinery  run  can  have  serious  consequences. 

If  the  conditional  distribution  of  y  is  to  be  approximately  norial,  then 


there  must  be  room  for  two  standard  deviations  between  and  the  nearest  bound. 
Therefore,  the  probability  of  large  values  of  must  be  kept  small.  Of  course, 
the  finite  range  of  y  puts  a  limit  on  the  size  of  q^  anyway,  but  the  normality 
assumption  snarpens  the  bound  considerably.  Plausible  skewed  conditional  dis¬ 
tributions  for  Y  could  have  values  of  much  closer  to  the  upper  bound.  The 
added  constraint  that  the  assumption  of  normality  places  on  plausible  values 
of  0^  is  more  stringent  cor  values  of  near  0  or  ir/2  than  for  near  ir/4  , 
so  th‘  ssumptlon  that  and  are  Independently  distributed  further  exacerbates 
the  problem. 

In  the  Novlck,  Lewis,  and  Jackson  procedure  the  estimated  proportions  for 
the  groups  are  obtained  from  the  mode  of  the  Joint  posterior  distribution  of  the 
y's  .  It  is  necessary  to  solve  a  set  of  simultaneous  equations,  the  Lindley 
equations,  to  find  this  mode.  The  solution  cannot  be  given  in  closed  form  and 
the  process  of  successive  approximation  can  require  many  iterations  to  converge. 
Sometimes  a  unique  mode  does  not  exist.  Novlck,  Lewis,  and  Jackson  give  formulas 


which  can  be  calculated  to  check  for  uniqueness. 

It  is  not  clear  how  critical  the  problems  Just  discussed  arc  in  practice. 
However,  these  problems  and  the  Increased  computational  complexity  of  the  proce¬ 
dure  do  suggest  that  other  ways  to  modify  Jackson's  simpler  approach  which  could 
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avoid  Its  weak  points,  but  retain  Its  computational  simplicity,  would  be  welcome. 
The  following  sections  will  show  that  simply  using  a  better  estimator  of  In 
Jackson's  procedure  goes  far  In  this  direction  and  that  an  even  simpler  approach 
Inspired  by  the  standard  Bayesian  analysis  for  a  single  proportion,  which  does 
not  require  a  transformation.  Is  possible.  The  latter  approach  Is  described  in 
the  next  section.  Jackson's  approach  and  the  proposed  modification  of  It  are 
discussed  In  the  section  after  that.  Finally,  some  examples  discussed  by  Novlck, 
Lewis,  and  Jackson  are  reexamined  In  the  last  section. 


5 

Estlmacion  of  Proportions 


2.  A  Beta-Blnomlal  Solution 

The  usual  Bayesian  approach  to  the  estimation  of  a  binomial  proportion 
£  assumes  that  the  Investigator's  personal  probability  distribution  for 
£  prior  to  the  gathering  of  data  can  be  represented  by  a  member  of  the 
beta  family  of  distributions,  the  natural  conjugate  family  for  the  binomial 
distribution.  If  we  let  _r'  and  n'  be  the  parameters  Identifying  the  par¬ 
ticular  member  of  this  family  which  represents  the  Investigator's  prior 
belief,  then  the  density  of  the  prior  distribution  is  given  by 


(2.1) 


B(r' ,n'  -  r 


(l-Z)*" 


-r'-l 


where  B(jr',  n'  -  is  the  complete  beta  function.  The  mean  of  the  investi¬ 
gator's  prior  distribution  is  £'/n'  and  the  posterior  distribution,  given 
£  successes  in  a  sample  of  £  tries  of  an  experiment  carried  out  to  provide 
further  Information  about  £,  is  the  beta  distribution  with  parameters  £  +  £* 
and  n  +  n'  .  The  mean  of  the  posterior  distribution,  which  can  be  used  as 
a  point  estimate  of  £  ,  is  given  by 


(2.2) 


'  r  +  r' 
^  n  +  n' 


n  .  ^  n'  .  _r^ 

n  +  n'  n  n  +  n'  n' 


The  second  expression  for  p  shows  that  it  is  a  weighted  average  of  the 
sample  proportion,  £/n  ,  and  the  mean  of  the  prior  distribution  of  £i£Vn'  . 
The  sample  proportion  receives  most  of  the  weight  if  £  is  large  rol.ative 
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to  n'  ,  whereas  the  overall  mean,  r_’/n'  >  receives  tlie  most  weight  if  n 
is  small  relative  to  n'  . 

When  data  from  J  different  groups  are  available,  it  is  possible  for 
the  experimenter  to  avoid  the  difficult  problem  of  specifying  his  or  her 
prior  distribution  for  £  ,  because  the  parameters  r_'  and  ti'  can  bo  esti¬ 
mated  from  the  data  (hence  the  term  empirical  Bayes) .  If  all  J  proportions 
are  based  on  the  same  number  of  observations,  and  the  J  different  £  parameters 
are  assumed  to  be  a  random  sample  from  a  population  with  a  beta  distribution, 
then  the  distribution  of  the  sample  proportions  is  beta-binomial.  The  parame¬ 
ters  of  the  beta-binomial  distribution  are  the  number  of  observations  per 
group,  the  number  of  groups,  and  the  parameters  £'  and  n'  of  the  underlying 
beta  distribution.  By  equating  the  sample  mean  and  variance  of  the  obtained 
number  of  successes  in  each  group  with  theoretical  expressions  for  their 
expected  values  in  a  beta-binomial,  one  obtains  a  system  of  equations  which 
can  be  solved  for  n'  and  r' ,  The  resulting  estimators  are 

^  n(n  -  1)  £.  (1  -  £.)  -  Av 
(2.3)  n'  - - > 

AVr 

r'  =  £.n'  , 

2 

where  £.  is  the  mean  of  the  sample  proportions  and  Av^  -  s^  -  ££.(1  -  £■ ) 
is  the  difference  between  the  sample  variance  of  the  nximber  of  successes 
between  groups  and  the  variance  one  would  expect  if  the  average  p  value 
had  generated  the  data  in  all  the  groups. 

Substituting  the  estimates  of  £*  and  £*  given  in  Equation  2.3  into 
Equation  2.2  yields  a  formula  for  estimating  the  proportions  in  each  group. 
These  results,  which  assume  that  the  sample  sizes  are  equal,  wore  given  by 
Marltz  (1966).  We  now  propose  a  generalization  which  docs  not  require  this 
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assumption.  First,  let  us  rewrite  the  expression  for  n'  in  Equation  2.3  as 


(2.4) 


£.  (1  -  £.)  -  Op2 


P 


where  a  ^  =  4v  /n  (n  -  1)  is  an  unbiased  estimate  of  the  variance  of  the 
P  ^ 

beta  distribution  generating  the  £'s.  The  idea  of  estimating  n'  and  £*  by 
equating  the  first  two  sample  moments  of  the  distribution  of  the  number  of 
successes  with  the  theoretical  values  for  the  beta-binomial  breaks  down 
because  the  sampling  distribution  is  no  longer  beta-binomial  when  the  £'s 
are  unequal.  Equation  2.4  can  be  used  as  the  basis  for  the  estimation  if 
we  can  find  a  suitable  estimator  for  .  One  possible  estimator  can  be 

obtained  from  a  general  theorem  concerning  the  analysis  of  variance  com¬ 
ponents  in  an  unbalanced  one-way  random  effects  model.  If  we  assume  the 
means  in  such  a  model  represent  a  random  sample  from  a  population  with 
mean  y  and  variance  ,  then,  even  without  making  the  assumption  of 
horaoscedasticity,  we  know  that 


(2.5) 


=  (J  -  1)  (n 


i:nj2\  -1 

-ff-  J  (MSjjg 

tween 


^^within^ 


is  an  unbiased  estimator  of  ,  where  n.  is  the  number  of  observations  in 

m  j 

group  j ,  N  is  the  total  number  of  observations  in  the  J  groups,  and 
^^between  ^^within  mean  sums  of  squares  in  an  unbalanced 

fixed  effects  analysis  of  variance. 

In  applying  the  theorem  to  the  problem  at  hand,  the  estimation  of 
Op  for  use  in  Equation  2.4  to  estimate  £*  and  jr'  ,  it  helps  to  note  that 
“  X^j  for  0-1  data.  The  sums  and  sums  of  squares  required  for  the 
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In  summary,  we^ propose  to  estimate  similar  proportions  in  J  different 
groups  as  follows.  First,  we  estimate  the  mean  of  the  distribution  of  the 
£  parameters  generating  the  proportions  by  £.  ■  R/N  and  the  variance  by 
Equation  2.7  .  Then  we  find  the  parameters  of  the  beta  distribution  having 
that  mean  and  variance  via  Equation  2.4  .  Adopting  this  beta  distribution 
as  our  prior,  we  apply  Equation  2.2  successively  to  the  sample  results  for 
each  group  to  obtain  the  posterior  expected  proportions  for  the  respective 


groups. 

In  the  case  of  equal  sample  sizes,  the  approach  Just  described  coincides 
with  one  proposed  by  Griffin  and  Krutekoff  (1971).  Tl\e  proposed  weighted 


“Stimator  of  for  the  extension  of  the  metl»od  to  Che  case  of  unequal  sample 
sizes  is  potentially  a  weak  link  in  the  procedure.  One  problem  is  that  it 


can  yield  negative  estimates  of  variance.  When  this  occurs,  we  assume  is 
zero  and  regress  the  estimates  of  the  proportions  for  all  groups  to  the  overall 


proportion  of  success.  It  will  be  argued  below  that  in  the  context  of  esti¬ 
mating  the  variance  of  proportions  between  groups  Che  proposed  estimator  is 
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less  subject  to  this  problem  than  unweighted  estimators  that  have  been 
suggested  and  is  satisfactory  for  the  use  Intended  here.  This  is  not  to 
say  that  it  is  the  best  possible  estimator.  The  problem  of  estimating  the 
between  group  variance  component  is  very  complicated  in  the  unbalanced  case. 
Tukey  (1957)  has  studied  the  variance  of  a  family  of  estimators  based  on  a 
very  general  weighting  scheme  that  includes  the  weighted  and  unweighted 
estimators  discussed  here  as  special  cases.  Which  of  the  latter  has 
smaller  variance  depends  in  a  complicated  way  on  several  factors.  Other 
estimators  in  the  family  can  be  devised  which  are  better  than  either  of 
these  under  some  circumstances.  A  single  "best"  estimator  does  not  exist. 

The  situation  seems  to  call  for  the  utilization  of  prior  information  about 
the  distribution  which  Bayesian  procedures  provide  for.  Hill  (1965)  has 
studied  such  procedures  under  normal  distribution  assumptions.  Unfortunately, 
these  procedures  are  cumbersome  to  apply  and  must  invoke  assumptions  and 
approximations  to  simplify  the  mathematics  which  make  one  question  how  much 
one  would  gain  by  employing  them.  The  legitimate  consideration  of  ease  of 
computation  will  often  limit  the  possibilities  seriously  entertained  to 
the  simple  weighted  and  unweighted  estimators.  As  has  already  been  indi¬ 
cated,  reasons  for  preferring  the  former  in  the  context  of  the  present 
problem  will  be  presented  below  in  the  discussion  of  application  of  the 
proposed  procedure. 

Before  discussing  applications  of  the  approach  to  estimation  of  propor¬ 
tions  in  several  groups  described  in  this  section,  let  us  describe  Jackson's 
(1972)  classical  Model  II  solution  to  the  problem.  This  approach  is  similar 
in  spirit  to  the  one  proposed  here,  but  somewhat  more  complicated  to  carry  out. 
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3.  An  Improved  Classical  Solution 

The  main  way  in  which  the  classical  Model  11  approacli  to  the  problem  of 
estimating  proportions  in  several  groups  differs  from  the  approach  Just 
described  is  Chat  it  first  transforms  the  proportions  via  the  root  arcsine 
transformation 

,  /r.  +  3/8\  h 


The  steps  carried  out  above  on  the  proportions  themselves  are  carried  out 
on  the  transformed  proportions  in  the  classical  approach.  The  transformed 
proportions  have  the  desirable  property  of  being  approximately  normally 
distributed  with  known  variances.  The  approxim.nte  mean  and  variance  of 
are  given  by 


(3.2) 


E(gj)  =  sin  =  Yj 


Var(g.) 


J  4nj  +  2 


where  is  the  true  proportion  in  group  j.  Thus,  the  variance  of  a  trans¬ 
formed  sample  proportion  depends  only  on  the  sample  size,  not  on  the  true 
value  of  the  proportion.  Let  and  be  the  mean  and  variance  of  the 
population  of  transformed  true  proportions  from  which  the  Yj's  are  considered 
to  have  been  sampled.  These  are  estimated  from  the  data  by  methods  to  be 
discussed  below.  Then  assuming  a  normal  prior  with  these  parameter  values. 
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the  posterior  means  of  the  transformed  proportions  are 


gi  +  u,  (4n^  +  2) 


+  (4nj  +  2)' 


When  theYj's  have  been  obtained,  the  estimated  proportions  themselves  are 


given  by  the  inverse  transformation 


=  sin'^  y. 

3  j 


Jackson  (1972)  estimates  the  mean  and  variance  of  the  transformed 


parameters  by 


U  -  Egj/J 

y  j  J 


o2  =  (J  -  1)“-^  Kg 
Y  j  J 


-  J-^  E(4nj  +  2)~^ 


That  is,  he  uses  the  unweighted  mean  of  the  transformed  sample  proportions 
as  the  estimator  of  the  mean,  and  the  difference  between  the  sample  variance 
of  the  observed  g.'s  and  the  mean  of  their  theoretical  variances  as  the 
estimator  of  the  variance  of  the  parameter  y  .  We  will  use  the  following 
weighted  estimators  instead: 


(3.6) 


(4N  +  2J)  r(4n^  +  2)gj  -  g.  , 


5<‘nj  +  2)2  ,-l  - 

(4N  +  2J  -  — - - 1  (z  (4n, 

\  4N  +  2J  /  \j  j 


+  2)(Rj  -  g.)-"  -  (J 


12 


Estimation  of  Proportions 

The  estimator  il^  weights  each  Inversely  to  Its  sampling  variance.  For  each 
j ,  the  standardized  variable 

Zg  -  +  2)^  (gj  -  Yj) 

Is  approximately  normal  In  distribution,  with  mean  0  and  variance  1.  It 
follows  that 

z2  .  (4nj  +  2)  (gj  -  Yj)2 

Is  approximately  distributed  as  a  chl-squared  variate  with  one  degree  of  freedom. 
If  there  Is  no  variation  In  theYj's,  then  the  statistic 

H  -  I(4n.  +2)  (g.  -  g.)2 

i  ^  ^ 

is  approximately  distributed  as  chi-squared  with  J-1  degrees  of  freedom.  In 
general,  its  expected  value  is  given  by 

I(4nj  +  2)2  \ 

E(H)  -  J  -  1  +  f4N  +  2J  -  J - )  • 

The  latter  relation  Is  the  reason  we  suggest  the  estimator  ^  defined  in 
:;:quatlon  3.5  .  The  weighted  estimators  In  Equation  3.5  reduce  to  the  esti¬ 
mators  proposed  by  Jackson  (1972)  when  the  sample  sizes  are  equal.  They  have 
much  less  sampling  variability  than  the  unweighted  estimators  when  the  sample 
sizes  are  grossly  unequal  and  is  relatively  small,  because  deviations  from 
the  overall  proportion  based  on  small  samples,  which  tend  to  be  large,  are 
given  less  weight  than  the  more  stable  proportions  based  on  larger  samples. 

It  will  be  seen  below  that  the  weighted  estimators  can  yield  quite  different 
results  than  the  unweighted  estimators. 
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4.  Reevaluation  of  Some  Applications 

Novick,  Lewis,  and  Jackson  (1973)  present  detailed  analyses  of  four  appli¬ 
cations  drawn  from  educational  settings.  The  last  example  In  their  discussion 
is  critical  because  it  Is  the  one  In  which  the  classical  empirical  Bayes 
approach  and  their  purely  Bayesian  approach  appear  to  be  most  clearly  contrasted. 
The  data  concern  the  performance  of  students  at  an  Eastern  University  who  were 
admitted  despite  the  fact  that  a  regression  analysis  based  on  Scholastic  Aptitude 
Test  scores  Indicated  that  they  would  be  unlikely  to  maintain  a  grade  point  aver¬ 
age  sufficient  to  complete  their  undergraduate  program.  The  students  were 
carefully  selected  on  the  basis  of  other  variables  such  as  demonstrated  moti¬ 
vation  and  willingness  to  work  hard  over  an  extended  period  of  time.  The 
question  addressed  by  the  data  is  whether  there  are  meaningful  differences  in 
success  rate  as  a  function  of  major  field  within  the  humanities  and  social 
sciences.  The  data  are  given  in  Table  1. 

When  one  applies  the  variance  estimator  given  in  Eq.  3.5  to  the  data  In 
Table  1,  the  resulting  estimate  of  the  variance  of  transformed  proportions  is 
-.004.  Given  this  result,  the  only  sensible  course  seems  to  be  to  assume  that 
■“here  is  no  real  variation  in  the  parameter.  A  purely  Bayesian  analysis,  on 
the  other  hand,  does  suggest  a  small  amount  of  real  variation  in  the  parameter, 
which  Novick  et  al.  argue  is  a  more  plausible  interpretation  of  the  data. 

One  might  agree  that  the  latter  interpretation  Is  the  more  plausible, 
but  on  the  grounds  that  the  negative  estimate  of  parameter  variance  reflects 
the  inadequacy  of  the  unweighted  estimator  of  ,  rather  than  the  Inadequacy 
of  the  classical  approach  per  se.  The  weighted  estimator  given  In  Eq.  3.6  yields 
an  estimate  of  of  +.006.  To  put  this  result  in  a  familiar  metric,  the  corres¬ 
ponding  chi-squared  test  statistic  for  testing  the  homogeneity  of  the  transformed 
proportions  Is  20.93,  which  almost  reaches  the  .05  level  critical  value  of  21.03. 
In  fact,  the  chl-squared  statistic  calculated  on  Che  untransformed  proportions 
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Table  1.  Comparison  of 
major  field.* 

different 

estimates 

Major  Field 

nj 

nj  + 

1 

10 

5 

.50 

2 

8 

6 

.75 

3 

72 

39 

.54 

4 

5 

4 

.80 

5 

25 

13 

.52 

6 

13 

11 

.85 

7 

81 

63 

.77 

8 

97 

72 

.74 

9 

21 

14 

.67 

10 

80 

62 

.78 

11 

11 

7 

.64 

12 

6 

5 

.83 

13 

8 

5 

.63 

of  success  rate  as  a  function  of 


B 

Pj 

C' 

C 

.68 

.66 

.66 

.70 

.69 

.71 

.71 

.70 

.64 

.59 

.60 

.70 

.70 

.71 

.71 

.70 

.71 

.62 

.63 

.70 

.71 

.74 

.73 

.70 

.72 

.76 

.75 

.70 

.71 

.73 

.73 

.70 

.69 

.69 

.69 

.70 

.72 

.75 

.75 

.70 

.69 

.69 

.69 

.70 

.70 

.72 

.72 

.70 

.69 

.69 

.69 

.70 

*Data  adapted  from  Novlck,  Lewis,  and  Jackson  (1973),  Table  VII. 

Column  B  gives  Bayesian  Model  II  estimates  for  the  prior  specified  by  Novlck 
et  alj^  (1973). 

Column  pj  gives  the  beta  binomial  estimates  described  in  Section  2. 

Column  C'  gives  the  estimates  using  the  revised  version  of  Jackson's  (1972) 
classical  method,  with  weighted  estimates  of  and  Oy  . 

Column  C  gives  the  classical  estimates  using  unweighted  estimators  of  Vy  and  Oy  . 

p  ”  .70  ,  Op  =  .006  (weighted  estimates). 

Y  »  .99  ,  °y  ”  .006  (weighted  estimates). 
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is  21.36,  which  would  lead  one  to  reject  homogeneity  at  the  .05  level.  Estimates 
of  parameter  variability  close  to  0  should  indicate  that  the  observed  variability 
between  samples  is  close  to  the  expected  value,  given  homogeneous  parameters, 
which  is  patently  not  the  case  with  these  data.  The  sample  sizes  are  extremely 
variable  in  this  example,  with  the  four  largest  samples  having  72  or  more  cases 
each  and  the  four  smallest  having  8  cases  or  less.  Any  adequate  estimator  of 
the  variation  between  groups  must  take  this  variation  of  sample  size  into  account 
The  weighted  estimator  of  the  variance  of  will  yield  negative  estimates 
only  when  the  variation  between  sample  proportions  is  such  that  the  chi-squared 
statistic  for  testing  equality  of  the  proportions  is  less  than  its  expected  value 
under  the  null  hypothesis.  This  is  analogous  to  a  F-statistlc  less  than  1  in 
testing  equality  of  means  in  an  analysis  of  variance.  Under  these  circumstances, 
the  sensible  thing  to  do  is  usually  to  use  the  overall  proportion  as  the  common 
estimator  for  all  of  them.  To  do  so  is  not  inconsistent  with  the  belief  that 
there  must  be  some  real  differences  between  the  parameters.  It  merely  indicates 
that  either  the  samples  are  so  small  or  the  sample  proportions  so  close  to  one 
another,  or  both,  that  distinct  estimates  for  each  proportion  would  be  misleading 
regarding  the  differences. 


In  order  to  make  sure  that  it  is  not  just  the  luck  of  the  draw  that  makes  the 
weighted  estimator  come  out  better  in  this  case,  one  hundred  replications  mim¬ 


icking  the  situation  in  this  data  set  were  simulated  on  a  computer.  In  each 
replication,  y's  having  mean  1.00  and  variance  .006  were  generated  for  thirteen 
groups  using  a  random  normal  number  generator.  The  7  for  each  group  was  then 


modified  by  a  random  normal  sampling  error  with  mean  0  and  variance  (4nj  +  2)”^ 
Then  weighted  and  unweighted  estimates  of  were  calculated.  The  distribution 


of  the  results  is  summarized  in  Table  2. 
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Table  2.  Distribution  of  weighted  and  unweighted  estimates  of  'n  100 
simulated  replications  under  conditions  represented  in  Table  1. 


Estimated 

ii 

Weighted 

Estimates 

Unweighted 

Estimates 

est . 

o2 

< 

.000 

10 

35 

.000  < 

tt 

< 

.003 

24 

11 

.003  < 

II 

< 

.006 

20 

8 

.006  < 

It 

< 

.009 

20 

11 

.009  < 

It 

< 

.012 

17 

10 

.012  < 

11 

_ 9 

25 

100 

100 

mean 

,00575 

.00567 

standard 

deviation 

.00527 

.OHIO 
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The  unweighted  estimates  are  negative  35  times  out  of  100,  wliereas  the 
weighted  estimates  are  negative  only  10  times  in  100.  The  unweighted  estimates 
are  more  than  double  the  actual  value  of  25  times  compared  to  only  9  times 
for  the  weighted  estimates.  Clearly,  under  the  conditions  present  in  this  data 
set,  namely  large  variation  in  sample  size  and  small  variation  in  y  ,  the  weighted 
estimator  is  much  better  behaved  than  the  unweighted  estimator.  These  remarks 
apply  with  equal  force  to  the  comparison  of  the  weighted  and  unweighted  estimators 
of  Op  in  the  direct  beta-binoeial  method  described  in  Section  2.  The  y 's  in  the 
simulations  just  described  were  transformed  to  p's  via  the  relation  p»sin^  , 
and  the  resulting  p's  used  to  generate  binary  data  for  each  group.  The  unweighted 
estimator  of  Op  yielded  negative  estimates  35  times,  whereas  the  weighted  esti¬ 
mator  in  Eq.  2.11  did  so  only  12  times.  The  standard  deviation  of  the  weighted 
estimator  was  less  than  half  that  of  the  unweighted  estimator. 

The  estimates  of  success  rate  by  major  field  produced  by  the  different 
estimators  are  compared  In  Table  1.  All  the  estimates  are  sharply  regressed 
toward  the  overall  success  rate  of  .70,  with  the  estimates  of  Jackson's  (1972) 
method  entirely  regressed  to  .70.  The  contrast  between  the  different  estimators 
is  clearest  for  Majors  3  and  5,  whose  observed  success  rates  of  .54  and  .52 
based  on  moderately  large  samples  represent  noteworthy  deviations  from  the  over¬ 
all  success  rate.  The  critical  ratios  for  these  deviations  from  a  hypothetical 
rate  of  .70  are  -2.93  and  -1.96  respectively.  For  Major  3  the  empirical  Bayes 
estimates  using  weighted  estimates  of  parameter  variation  are  .59  for  the  direct 
beta-binomial  method  and  .60  for  Jackson’s  method;  for  Major  5  these  methods  yield 
.62  and  .63,  respectively.  There  is  little  to  choose  between  the  direct  beta- 
binomial  method  and  the  revised  version  of  Jackson '.s  method  in  this  set  of  data, 
since  the  two  approaches  never  differ  by  more  than  one  point  in  the  second 
d  cimal  place.  The  estimates  produced  by  either  metliod  seem  more  plausible  for 
Majors  3  and  5  than  .70. 
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The  data  set  just  discussed  is  well  suited  for  showing  the  weighted  esti¬ 
mators  at  their  best  because  the  sample  sizes  arc  so  variable  and  the  apparent 
variation  in  the  underlying  proportions  so  small.  Let  us  turn  now  to  an  example 
more  likely  to  show  the  unweighted  estimators  at  tljolr  best.  The  first  data 
set  discussed  by  Novlck,  Lewis,  and  Jackson  (1973)  involved  the  analysis  of 
the  proportions  of  students  attaining  the  70th  percentile  on  the  Iowa  Test  of 
Educational  Development  composite  score  in  eleven  Midwestern  high  schools. 

Novlck  et  al.  consider  several  possible  specifications  of  prior  for  o^  ,  the 
one  with  largest  variance  having  mean  equal  to  .035  .  They  argue  that  this  is 
about  as  large  as  it  can  be  and  still  be  consistent  with  the  constraints  imposed 
by  the  desire  that  the  prior  for  have  an  inverse  chi-square  form  with  mode 
falling  within  the  interquartile  range,  and  the  fact  that  they's  are  in  the 
bounded  interval  (0,  ir/2).  The  sample  sizes  in  this  example  range  from  14  to 
30,  but  the  second  largest  sample  has  only  21  subjects,  so  the  variability  in 
sample  size  Is  much  less  extreme  than  It  was  in  the  previous  example.  Moderate 
variation  in  sacjple  size  and  large  variation  in  the  underlying  parameter  should 
make  this  example  very  favorable  to  unweighted  estimators  of  and  . 

A  simulation  study  similar  to  the  one  done  in  the  previous  example  was 
conducted  using  the  sample  sizes  of  the  present  example,  a  mean  y  of  .80  ,  and 
equal  to  .035  .  The  unweighted  and  the  weighted  estimates  of  were  negative 
in  3  of  100  replications  (the  same  replications,  as  it  happened).  The  standard 
deviation  and  mean  squared  error  were  slightly  smaller  for  the  unweighted  esti¬ 
mator,  .018  vs  .020  .  The  correlation  between  tlie  two  estimators  was  .974  . 

The  results  were  even  closer  for  the  two  estimators  of  .  The  weighted  esti¬ 
mate  was  negative  once  whereas  the  unweighted  estimates  never  were.  On  the 
other  hand  the  weighted  estimator  was  (very)  slightly  loss  variable  than  the 
unweighted  estimator. 
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The  results  of  the  simulation  study  suggest  that  it  will  make  little 

difference  whether  weighted  or  um-;eighted  estimators  are  used  when  there  is 

not  too  much  variability  In  sample  size  and  the  variation  in  the  underlying 

proportions  is  substantial.  The  analyses  of  data  for  the  example  confirm  this 

expectation.  The  data  are  given  in  Table  3.  Estimators  C,  C',  and  p  are 

defined  as  in  Table  1,  When  Novick  et  al.  applied  their  Bayesian  analysis  using 

a  diffuse  prior  with  ■  .035  ,  the  posterior  estimate  of  went  down  to  .015  . 

The  marked  discrepancy  between  prior  and  posterior  estimates  of  led  them  to 

Y 

try  another  analysis  using  the  posterior  specification  of  as  if  it  were 
their  prior.  The  results  of  this  analysis,  which  is  similar  in  spirit  to  the 
empirical  Bayes  approach,  are  given  in  the  column  headed  B(t“16).  The  various 
analyses  yield  very  similar  results.  The  largest  discrepancy  between  the 
weighted  and  unweighted  versions  of  Jackson's  method  is  .010  .  Between  the 
direct  beta^blnomlal  approach  and  the  weighted  version  of  Jackson's  method  the 
largest  discrepancy  is  .005  .  The  largest  difference  between  any  two  of  the 
methods  is  .014  . 

In  summary,  the  differences  in  performance  between  wclghtiJ  .md  unweighted 
estimators  of  q^  and  are  not  very  significant  when  q^  or  are  large  and 
the  variability  of  sample  sizes  is  moderate;  that  is,  under  conditions  which 
would  in  general  favor  unweighted  estimators.  On  the  other  hand,  the  differences 
in  favor  of  weighted  estimators  are  substantial  when  the  sample  sizes  are 
extremely  variable  and  is  small.  It  is  therefore  recommended  that  weighted 
estimators  be  used  as  a  general  practice  when  applying  these  procedures. 

The  advantages  of  the  empirical  Bayes  approaches  over  a  strictly  Bayesian 
Model  II  approach  to  the  estimation  of  proportions  in  several  groups  are  twofold: 
they  are  much  easier  to  compute  and  they  avoid  the  problem  of  the  specification 
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Table  3.  Comparison  of  analyses  using  different  estimators  of  proportions  of 

students  attaining  the  70th  percentile  on  the  Iowa  Test  of  Educational 
Development  composite  score  In  eleven  Midwestern  high  schools.* 


School 


Number 

nj 

Pj 

1 

15 

10 

.667 

2 

21 

13 

.619 

3 

16 

5 

.313 

A 

17 

10 

.588 

5 

15 

6 

.400 

6 

17 

11 

.647 

7 

18 

9 

.500 

8 

20 

15  - 

.750 

9 

30 

12 

.400 

10 

19 

8 

.421 

11 

14 

7 

.500 

C 

C' 

A 

Pj 

B(t»16) 

.555 

.557 

.561 

.562 

.551 

.552 

.555 

.556 

.481 

.473 

.468 

.468 

.541 

.540 

.543 

.543 

.502 

.496 

.493 

.494 

.554 

.556 

.559 

.560 

.521 

.518 

.518 

.518 

.585 

.591 

.595 

.599 

.488 

.478 

.474 

.474 

.502 

.496 

.493 

.494 

.522 

.519 

.519 

.520 

*Data  adapted  from  Novick,  Lewis,  and  Jackson  (1973),  Table  IV 
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of  a  prior  distribution.  The  discussion  of  the  latter  problem  by  Novick,  Lewis, 
and  Jackson  (1973)  shows  that  it  can  be  a  rathe*"  delicate  matter.  An  objection 
to  the  empirical  Bayes  methods  is  that  the  estimators  of  the  variance  of  the 
underlying  parameter  are  so  poor  that  they  cannot  be  trusted.  The  examples 
presented  here  suggest  that  the  use  of  weighted  estimators  reduces  the  force 
of  this  objection  considerably. 

The  weighted  estimators  are  by  no  means  perfect  and  in  some  other  contexts 
an  investigator  might  be  well  advised  to  use  a  Bayesian  estimator  enabling  him 
to  Incorporate  his  prior  beliefs  about  the  extent  of  variation  into  his  estimate. 
This  might  be  the  case,  for  example,  if  the  data  represent  the  results  of  a 
pilot  study  to  determine  the  size  of  sample  required  to  estimate  the  between 
group  variance  component  accurately.  The  practice  of  setting  a  negative  esti¬ 
mate  of  variance  to  zero  could  lead  to  a  rather  misleading  conclusion  in  such 
a  situation.  However,  in  the  present  context,  setting  a  negative  estimate  of 
variance  to  zero  leads  to  the  sensible  decision  to  regress  of  estimates  of 
proportions  to  the  overall  proportion.  For  present  purposes .  the  weighted 
estimators  proposed  in  this  paper  seem  to  be  adequate. 

The  beta-binomial  method  of  Section  2  and  Jackson's  classical  method 
yield  practically  identical  results  when  weighted  estimators  of  Op  and  o^ 
are  used,  at  least  for  the  examples  presented  here.  This  is  not  too  surpris¬ 
ing  since  the  root  arcsine  transformation  is  a  very  weak  transformation  of 
proportions  falling  in  the  broad  middle  range  of  the  unit  interval,  as  is  the 
case  with  the  examples  discussed  here.  In  this  range  it  in  effect  just  adds 
the  constant  it/4  -  *  .29  to  each  proportion,  where  the  constant  Is  the  differ¬ 

ence  between  the  midrange  of  transformed  unit  interval  and  the  midrange  of  unit 
interval  itself.  For  example,  the  interval  (.20,  .80)  is  transformed  to  the 
Interval  (.46,  1.11)  by  the  root  arcsine  transformation  and  to  (.49,  1.09)  by 
adding  the  constant  .29  .  Even  for  the  interv.nl  (.10,  .90),  the  transformed 
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interval  (.32,  1.25)  is  not  very  different  from  the  shifted  intcrv.nl  (.39,  1.19). 
The  root  arcsine  transformation  has  a  siibst.nit i.nl  effect  on  the  sp.icliig  of  pro¬ 
portions  only  when  they  are  concentrated  near  zero  or  one.  In  such  cases 
Jackson's  method,  using  the  Freeman  and  Tukey  (1950)  modification  of  the  arcsine 
transformation 


may  yield  better  results  than  the  beta-binomial  method.  However,  for  problems 
where  most  of  the  proportions  are  between  .10  and  .90  the  direct  bct.n-binomial 
approach  would  be  preferred  because  of  its  comput.ntional  and  conceptual 
simplicity. 
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